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Reduced chi-squared is a very popular method for model assessment, model comparison, 
convergence diagnostic, and error estimation in astronomy. In this manuscript, we discuss 
the pitfalls involved in using reduced chi-squared. There are two independent problems: 
(a) The number of degrees of freedom can only be estimated for linear models. Concerning 
nonlinear models, the number of degrees of freedom is unknown, i.e., it is not possible to 
compute the value of reduced chi-squared. (b) Due to random noise in the data, also 
the value of reduced chi-squared itself is subject to noise, i.e., the value is uncertain. 
This uncertainty impairs the usefulness of reduced chi-squared for differentiating between 
models or assessing convergence of a minimisation procedure. The impact of noise on the 
value of reduced chi-squared is surprisingly large, in particular for small data sets, which 
are very common in astrophysical problems. We conclude that reduced chi-squared can 
only be used with due caution for linear models, whereas it must not be used for nonlinear 
models at all. Finally, we recommend more sophisticated and reliable methods, which are 
also applicable to nonlinear models. 

1 Introduction 

When fitting a model / with parameters 9 to N data values y n , measured with (uncorrelated) 
Gaussian errors a n at positions x n , one needs to minimise 



This is equivalent to maximising the so-called "likelihood function" . If the data's measurement 
errors are not Gaussian, x 2 should not be used because it is not the maximum-likelihood 
estimator. For the rest of this manuscript, we shall therefore assume that the data's errors are 
Gaussian. If K denotes the number of degrees of freedom, reduced x 2 is then denned by 



Xred i s a quantity widely used in astronomy It is essentially used for the following purposes: 

1. Single- model assessment: If a model is fitted to data and the resulting x 2 e d ls larger than 
one, it is considered a "bad" fit, whereas if x 2 e d < 1, it is considered an overfit. 

2. Model comparison: Given data and a set of different models, we ask the question which 
model fits the data best. Typically, each model is fit to the data and their values of x 2 e d 
are compared. The winning model is that one whose value of x 2 e d is closest to one. 




(1) 



Xred 
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Convergence diagnostic: A fit is typically an iterative process which has to be stopped 
when converged. Convergence is sometimes diagnosed by monitoring how the value of 
Xred evolves during the iteration and the fit is stopped as soon as Xred reaches a value 
sufficiently close to one. Sometimes it is claimed then, that "the fit has reached noise 
level". 

Error estimation: One fits a certain model to given data by minimising y 2 and then 
rescales the data's errors such that the value of x 2 e d i s exactly equal to one. From this 
one then computes the errors of the model parameters. (It has already been discussed by 



Andrae (2010) that this method is incorrect, so we will not consider it any further here.) 



In all these cases, Xred excels in simplicity, since all one needs to do is divide the value of x 2 by 
the number of degrees of freedom and compare the resulting value of x 2 e d t° one. 

In this manuscript, we want to investigate the conditions under which the aforementioned 
applications are meaningful - at least the first three. In particular, we discuss the pitfalls that 
may severly limit the credibility of these applications. We explain the two major problems that 
typically arise in using x^d 111 practice: First, we dicuss the issue of estimating the number of 
degrees of freedom in Sect. [2j Second, we explain how the uncertainty in the value of x 2 may 
affect the above applications in Sect. |3j Sectionj|]is then dedicated to explain more reliable 
methods rather than x 2 e d- We conclude in Sect, 



2 Degrees of freedom 

Given the definition of Xred' ^ ^ s evidently necessary to know the number of degrees of freedom 
of the model. For iV data points and P fit parameters, a naive guess is that the number of 
degrees of freedom is iV — P. However, in this section, we explain why this is not true in 
general. We begin with a definition of "degrees of freedom" and then split this discussion into 
three parts: First, we discuss only linear models. Second, we discuss linear models with priors. 
Third, we discuss nonlinear models. Finally, we discuss whether linearisation may help in the 
case of nonlinear models. 



2.1 Definition 

For a given parameter estimate, e.g., a model fitted to data, the degrees of freedom are the 
number of independent pieces of information that were used. The concept of "degrees of 
freedom" can be defined in different ways. Here, we give a general and simple definition. In 
the next section, we give a more technical definition that only applies to linear models. 

Let us suppose that we are given N measurements y n and a model with P free parameters 
0i, 02, ■ ■ ■ , Op. The best-fitting parameter values are found by minimising x 2 ■ This means we 
impose P constraints of the type 

|£ = Vp=l,2,...,P (3) 

onto our N- dimensional system. Hence, the number of degrees of freedom is K = N — P. At 
first glance, this appears to be a concise and infallible definition. However, we shall see that 
this is not the case. 
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2.2 Linear models without priors 

A linear model is a model, where all fit parameters are linear. This means it is a linear 
superposition of a set of basis functions, 

p 

/(£, 9) = exB^x) + 9 2 B 2 {x ) + . . . + 6 P B P {x) = £ p S p (f) , (4) 

P =i 

where the coefficients 9 P are the fit parameters and the B p (x) are some (potentially nonlinear) 
functions of the position x. A typical example is a polynomial fit, where B p (x) = x p . Inserting 
such a linear model into \ 2 causes \ 2 to be a quadratic function of the fit parameters, i.e., 
the first derivatives - our constraints from Eq. (|3j) - form a set of linear equations that can be 
solved analytically under certain assumptions. 

A natural approach to solve such sets of linear equations is to employ linear algebra. This 
will lead us to a quantitative definition of the number of degrees of freedom for linear models. 
Let us introduce the following quantities: 

• V — {Vii V2i ■ ■ ■ j Un) T is the iV-dimensional vector of measurements y n . 

• 9 = (#i, 9 2 , ■ ■ ■ , 0p) T is the P-dimensional vector of linear model parameters 9 p . 

• S is the N x N covariance matrix of the measurements, which is diagonal in the case of 
Eq. Q, i.e., S = diag(a 1 2 , <rf , . . . , a%). 

• X is the so-called design matrix which has format N x P. Its elements are given by 
X np = B p (x n ), i.e., the p-th basis function evaluated at the n-th measurement point. 

Given these definitions, we can now rewrite Eq. ([T]) in matrix notation, 



X 



2 -{y-X-W-^-iy-X-e). (5) 



Minimising x 2 by solving the constraints of Eq. ^ then yields the analytic solution 

6 = {X T -Yr l -X)- x ■ X T -y , (6) 

where the hat in 9 accounts for the fact that this is only an estimator for 9, but not the true 9 
itself. We then obtain the prediction y of the measurements y by, 

fj = X ■ 9 = X ■ (X T ■ S" 1 ■ X)" 1 • X T ■ S" 1 • y = H ■ y , (7) 

where we have introduced the N x N matrix H, which is sometimes called "hat matrix", 
because it translates the data y into a model prediction y. The number of effective model 
parameters is then given by the trace of H (e.g. Ye 1998 Hastie et al. 2009), 

N 

P cS = tv{H) = Y,H nn = rank(X) , (8) 

n=l 

which also equals the rank of the design matrix X^ Obviously, P e fr < P, where the equality 
holds if and only if the design matrix X has full rank. Consequently, for linear models the 
number of degrees of freedom is 

K = N-P eS >N-P. (9) 
1 The rank of X equals the number of linearly independent column vectors of X . 
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The standard claim is that a linear model with P parameters removes P degrees of freedom 
when fitted to N data points, such that the remaining number of degrees of freedom is K = iV — 
P. Is this correct? No, not necessarily so. The problem is in the required linear independence 
of the P basis functions. We can also say that the P constraints given by Eq. ^ are not 
automatically independent of each other. Let us consider a trivial example, where the basis 
functions are clearly not linearly independent: 

Example 1 The linear model f{x,9) = 9\ + 9 2 is composed of two constants, 9i and 62, i.e., 
P>i(x) = B 2 (x) = 1. Obviously, this two-parameter linear model cannot fit two arbitrary data 
points and its number of degrees of freedom is not given by N — 2 but N — 1, because the design 
matrix X only has rank 1, not rank 2. In simple words, the two constraints = and = 
are not independent of each other. 

From this discussion we have to draw the conclusion that for a linear model the number of 
degrees of freedom is given by N — P if and only if the basis functions are linearly independent 
for the given data positions x n , which means that the design matrix X has full rank. In practice, 
this condition is usually satisfied, but not always. In the more general case, the true number 
of degrees of freedom for a linear model may be anywhere between N — P and N — 1. 



2.3 Linear models with priors 



Priors are commonly used to restrict the possible 
values of fit parameters. In practice, priors are 
usually motivated by physical arguments, e.g., a 
fit parameter corresponding to the mass of an 
object must not be negative. Let us consider a 
very simple example of a prior: 

Example 2 A linear model f(x,ao,ai) = a + 
a±x is given. The value of parameter oq is not 
restricted, but a% must not be negative. Figure 
[7] demonstrates that this two-parameter model is 
incapable of sensibly fitting two arbitrary data 
points because of this prior. 

Obviously, priors reduce the flexibility of a 
model, which is actually what they are designed 
to do. Consequently, they also affect the num- 
ber of degrees of freedom. In this case, the prior 
was a step function (zero for a\ < and one 
otherwise), i.e., it was highly nonlinear. Conse- 
quently, although f(x, ao, ai) = a + aix is itself 
a linear function of all fit parameters, the overall 
model including the prior is not linear anymore. 
This leads us directly to the issue of nonlinear models. 




Figure 1: Example of a two-parameter 
model, f(x) = a + a±x, that is incapable 
of fitting two data points perfectly because 
it involves a prior ai > 0. We show the op- 
timal fit for the given data. The model is 
discussed in Example [2} 



2.4 Nonlinear models 



We have seen that estimating the number of degrees of freedom is possible in the case of linear 
models with the help of Eq. However, for a nonlinear model, we cannot rewrite \ 2 like in 
Eq. ([5]), because a nonlinear model cannot be written as X ■ 9. Therefore, H does not exist and 
we cannot use Eq. d8|) for estimating the number of degrees of freedom. Ye (1998) introduces 
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the concept of "generalised degrees of freedom", but concludes that it is infeasible in practice. 
We now consider two examples in order to get an impression why the concept of degrees of 
freedom is difficult to grasp for nonlinear models: 

Example 3 Let us consider the following model f(x), having three free parameters A, B, C , 

f(x) = Acos(Bx + C) . (10) 

If we are given a set of N measurement (x n ,y n ,o~ n ) such that no two data points have identical 
x n , then the model f(x) is capable of fitting any such data set perfectly. The way this works is 
by increasing the "frequency" B such that f(x) can change on arbitrarily short scales^ As f(x) 
provides a perfect fit in this case, \ 2 ^ s equal to zero for all possible noise realisations of the 
data. Evidently, this three-parameter model has infinite flexibility (if there are no priors) and 
K = N — P is a poor estimate of the number of degrees of freedom, which actually is K = 0. 

Example 4 Let us modify the model of Example [3] slightly by adding another component with 
additional free parameters D, E, and F , 

f(x)=Acos(Bx + C)+Dcos(Ex + F). (11) 

If the fit parameter D becomes small such that \D\ <C \A\, the second component cannot influence 
the fit anymore and the two model parameters E and F are "lost". In simple words: This model 
may change its flexibility during the fitting procedure. 

Hence, for nonlinear models, K may not even be constant^ Of course, these two examples 
do not verify the claim that always K ^ N — P for nonlinear models. However, acting as 
counter-examples, they clearly falsify the claim that K = N — P is always true for nonlinear 
models. 



2.5 Local linearisation 

As we have seen above, there is no well-defined method for estimating the number of degrees 
of freedom for a truly nonlinear model. We may now object that any well-behaved nonlinear 
modeQcan be linearised around the parameter values which minimise \ 2 ■ Let us denote the 
parameter values that minimise x 2 by B. We can then Taylor-expand x 2 at @ to second order, 



X 2 (0)^xLn+ £ 



. d0 p d0 q 

p,q=l y H 



(9 p -e p )(e q -e q ), (12) 

9=0 



where the first derivative is zero at the minimum. Apparently, \ 2 is now a quadratic function of 
the model parameters, i.e., the model is linearised. Does this mean that we can simply linearise 
the model in order to get rid of the problems with defining the number of degrees of freedom 
for a nonlinear model? 

The answer is definitely no. The crucial problem is that linearisation is just an approxima- 
tion. The Taylor expansion of Eq. (12) has been truncated after the second-order term. There 



2 In practice, there is of course a prior forbidding unphysically large frequencies. But there is no such 
restriction in this thought experiment. 

3 For linear models this cannot happen, since products (or more complicated functions) of model parameters 
are nonlinear. 

4 With "well-behaved" we mean a model that can be differentiated twice with respect to all fit parameters, 



including mixed derivatives. 
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are two issues here: First, in general, we do not know how good this approximation really is for 
a given data sample. Second, we have no way of knowing how good the approximation needs 
to be in order to sufficiently linearise the model from the number-of-degrees-of-freedom point 
of view. 

Even if these issues did not concern us, would linearising the model really help? Again, 
the answer is no. As we have seen in Sect. 2.2, the number of degrees of freedom is also not 



necessarily given by N—P for a linear model. Moreover, the truly worrisome result of Sect. 2.4 



that the number of degrees of freedom is not constant - is not overcome by the linearisation. The 
reason is that the expansion of Eq. ( 12 ), and thereby the linearisation, depends nonlinearly upon 
where the maximum isj^] Consequently, the uncertainties in the maximum position inherited 
from the data's noise propagate nonlinearly through the expansion of Eq. (12). Therefore, we 
have to draw the conclusion that there is no way of reliably estimating the number of degrees 
of freedom for a nonlinear model. 



2.6 Summary 

Summarising the arguments brought up so far, we have seen that estimating the number of 
degrees of freedom is absolutely nontrivial. In the case of linear models, the number of degrees 
of freedom is given by N — P if and only if the basis functions are indeed linearly independent 
in the regime sampled by the given data. Usually, this is true in practice. Otherwise, the 
number of degrees of freedom is somewhere between N — P and N — 1. However, in the case 
of nonlinear models, the number of degrees of freedom can be anywhere between and N — 1 
and it is even not necessarily constant during the fit. Linearising the model at the optimum 
does not really help to infer the number of degrees of freedom, because the linearised model 
still depends on the optimum parameters in a nonlinear way. Hence, it is questionable whether 
it is actually possible to compute x 2 e d f° r nonlinear models in practice. 

3 Uncertainty in % 2 

We now discuss another problem, which is completely independent of our previous considera- 
tions. Even if we were able to estimate the number of degrees of freedom reliably, this problem 
would still interfere with any inference based on Xred- This problem stems from the fact that 
the value of x 2 is subject to noise, which is inherited from the random noise of the dataj^] 
Consequently, there is an "uncertainty" on the value of x 2 an d hence on xLdi which is typically 
ignored in practice. However, we show that this uncertainty is usually large and must not be 
neglected, because it may have a severe impact on the intended application. 

Given some data with Gaussian noise, the true model having the true parameter values will 
generate a y 2 = N and has N degrees of freedom because there is no fit involved. Hence, it 
results in a x 2 e d °f 1- We therefore compare the Xred °f our trial model to 1 in order to assess 
convergence or to compare different models. Is this correct? 

In theory, yes. In practice, no. Even in the case of the true model having the true parameter 
values, where there is no fit at all, the value of \ 2 is subject to noise. In this case, we are 
fortunate enough to be able to quantify this uncertainty. For the true model having the true 

5 For a linear model the second derivatives of y 2 do not depend on any model parameters, i.e., they are 
constant. 

6 For a given set of data, \ 2 can of course be computed. However, consider a second set of data, which was 
drawn from the same physical process such that only the noise realisation is different. For this second set of 
date, the value of \ 2 wm differ from that of the first set. 
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parameter values and a-priori known measurement errors <r n , the normalised residuals, 



a„ 



(13) 



are distributed according to a Gaussian with mean /x = and variance a 2 = lQ In this case 
only, x 2 is the sum of K = N Gaussian variates and its probability distribution is given by the 
so-called ^-distribution, 



prob(x 2 ; K) 



2K/2 T (K/2) 



(x 2 ) 



(14) 



Figure [2] shows some ^-distributions with different values of K . The expectation value of the 
X 2 -distribution is, 



(x 2 ) 



0.030 



X 2 wob( X 2 ;K)d x 2 = K . (15) 



In fact, this expectation value is sometimes used 
as an alternative definition of "degrees of free- 
dom". As the ^-distribution is of non-zero 
width, there is however an uncertainty on this 
expectation value. More precisely, the variance 
of the ^-distribution is given by 2K. This 
means the expectation value of x 2 e d f° r the true 
model having the true parameter values is in- 
deed one, but it has a variance of 2/K = 2/N. 
If iV is large, the ^-distribution becomes ap- 
proximately Gaussian and we can take the root 
of the variance, o = y2~jN \ as an estimate of 
the width of the (approximately Gaussian) peak. 
Let us consider a simple example in order to get 
a feeling how severe this problem actually is: 




0.000, 



Figure 2: ^-distributions for different values 
of K = N degrees of freedom. The distribu- 
tions are asymmetric, i.e., mean and maxi- 
mum (mode) do not coincide. For increasing 
K = N, the distributions become approxi- 
mately Gaussian. 



Example 5 We are given a data set comprised 
of N = 1, 000 samples. Let the task be to use x 2 e d 

in order to compare different models to select that one which fits the data best, or to fit a single 
model to this data and assess convergence. The true model having the true parameter values - 
whether it is given or not - will have a value of ' x 2 e d w ^ an (approximated) Gaussian standard 
deviation of a = a/2/1000 ~ 0.045. Consequently, within the 3cr interval 0.865 < xled — 1-135 
we can neither reliably differentiate between different models nor assess convergence. 

This simple example clearly shows that this problem is very drastic in practice. Moreover, 
astronomical data sets are often much smaller than N = 1, 000, which increases the uncertainty 

ofx r 2 ed- 

Of course, there is not only an uncertainty on the comparison value of x 2 e d l° r the true 
model having the true parameter values. There is also an uncertainty on the value of x 2 e & 
for any other model. Unfortunately, we cannot quantify this uncertainty via a w sj2jK in 



7 Again, we implicitely assume that the Gaussian errors are uncorrelated, as in Eq. Q. If the measurement 
errors a n are not known a priori but have been estimated from the data, the normalised residuals R n are drawn 
from Student's i-distribution (e.g. Barlow|1993 ). With increasing number of data points Student's i-distribution 
approaches a Gaussian distribution. 
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practice anymore, because the ^-distribution applies only to the true model having the true 
parameter values. For any other model the normalised residuals (cf. Eq. (13)) are not Gaussian 
with mean fi = and variance a 2 = 1. Hence, \ 2 is not the sum of K Gaussian variates and 
the derivation of the ^-distribution is invalid. 



4 Alternative methods 

If Xred does not provide a reliable method for assessing and comparing model fits, convergence 
tests or error estimation, what other methods can then be used with more confidence? An 
in-depth survey of alternative methods would be beyond the scope of this manuscript. There- 
fore, we restrict our discussion on some outstanding methods. Concerning methods for error 



estimation, we refer the interested reader to Andrae (2010). 



4.1 Residuals 

The first and foremost thing to do in order to assess the goodness of fit of some model to some 
data is to inspect the residuals. This is indeed trivial, because the residuals have already been 
computed in order to evaluate \ 2 (cf- Eq. 0). For the true model having the true parameter 
values and a-priori known measurement errors, the distribution of normalised residuals (cf. Eq. 
(13)) is by definition Gaussian with mean \x = and variance a 2 = 1. For any other model, this 
is not true. Consequently, all one needs to do is to plot the distribution of normalised residuals 
in a histogram and compare it to a Gaussian of /i = and o 2 = 1. If the histogram exhibits 
a statistically significant deviation from the Gaussian, we can rule out that the model is the 
truth. If there is no significant difference between histogram and Gaussian, this can mean (a) 
we found the truth, or (b) we do not have enough data points to discover the deviation. The 
comparison of the residuals to this Gaussian should be objectively quantified, e.g., by using a 
Kolmogorov-Smirnov test|^ ( Kolmogorov 1933 Smirnov 1948). 

In theory, this method may be used as a convergence diagnostic. In an iterative fit procedure, 
compare the distribution of normalised residuals to the Gaussian with fi = and a 2 = 1 in 
each iteration step, e.g., via a Kolmogorov-Smirnov test. At first, the likelihood of the residuals 
to be Gaussian will increase as the model fit becomes better. If the fit finds a suitable local 
minimum, the model may eventually start to overfit the data and the likelihood of the residuals 
to be Gaussian will decrease again, as the residuals will peak too sharply at zero. When this 
happens, the fitting procedure should be stopped. In practice, there is no guarantee that this 
works, as the fit may end up in a local minimum with residuals too widely spread to resemble 
the Gaussian with \x = and a 2 = 1. 

Similarly, this method may also be used for model comparison. Given some data and a 
set of models, the model favoured by the data is that whose normalised residuals match the 
Gaussian with /i = and o 2 = 1 best. The winning model does not need to be the truth. Let 
us consider the following example: 



Example 6 Vogt et al. (2010) analysed radial-velocity data of the nearby star GJ 581 and came 
to the conclusion that the data suggests the presence of six exoplanets instead of four as claimed 
by other authors using different data (e.g. Mayor et al. 2009). Vogt et al. (201Cfy assumed 
circular orbits, which result in nonlinear models of the form of Eq. (10) in Example^ Their 
claim that two additional planets are required is primarily justified from the associated x 2 re d ( c f- 



8 The Kolmogorov-Smirnov (KS) test compares the empirical cumulative distribution function (CDF) of a 
sample to a theoretical CDF by quantifying the distance between the distributions. Under the (null) hypothesis 
that the sample is from the given distribution, this distance (called the KS-statistic) has a known probability 
distribution. Now, the test calculates the KS-statistic and compares it to its known probability distribution. 
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Planets 1 
p- value 8.71 ■ 10" 11 
X r 2 ed 8.426 



2 3 

2.97 ■ 10" 9 2.51 ■ 10~ 7 
4.931 4.207 



1.28 • 10- 4 

3.463 



1.47- 10" 5 
2.991 



6 

6.97- 10- 
2.506 



Table 1: p- values from KS-test and x? e d f° r 1~6 planets for the data of Vogt et al. (2010) 
discussed in Example |6j 



Table^Q). We take the identical data as Vogt et al. (201Cfy and their asserted orbital parameters 
of the six planets (their Table 2). For every number of planets, we apply the KS-test to the 
normalised residuals. Table^ gives the resulting p-valueM for the six models. In terms of the 
KS-test, the data used by Vogt et al. ( 2010]) strongly favour the model using four planets over the 
model using six planets. Furthermore, Fig. [3] displays the distributions of normalised residuals 
for the models using four and six planets. Evidently, both distributions deviate significantly from 
the Gaussian with \x = and a 2 = 1, which implies that neither model is compatible with the 
truth. The most likely explanation for this discrepancy is that planetary orbits may be elliptical, 



whereas Vogt et al. (2010) assumed circular orbits 



There is a pitfall here, as well. The likelihood of the normalised residuals to come from a 
Gaussian with \i = and a 2 = 1 is also subject to noise, as in the case of the value of \ 2 - 
However, these uncertainties surely cannot explain the large differences in Table [T] 



4.2 Cross-validation 

Cross-validation is one of the most powerful and 
most reliable methods for model comparison. 
Unfortunately, it is usually also computationally 
expensive. However, if Xred * s n °t applicable, 
e.g., because the model is nonlinear, computa- 
tional cost cannot be used as an argument in 
disfavour of cross-validation. 

The most straightforward (and also most ex- 
pensive) flavour of cross-validation is "leave-one- 
out cross-validation". We are given N data 
points and a set of models, and we want to know 
which model fits the data best. For each model, 
the goodness of fit is estimated in the following 
way: 

1. Remove the n-th data point from the data 
sample. 

2. Fit the model to the remaining N — 1 data 
points. 

3. Take the model fitted to the N — 1 data 
points and compute its likelihood for the 
n-th data point that has been left out. 

9 Loosely speaking, in this case, the p- value is the probability that the true model can generate residuals that 
agree with the standard Gaussian as badly as or worse than the actually observed ones. 




-2 2 

normalised residuals 



Figure 3: Distributions of normalised resid- 



uals for the data and models of Vogt et al. 



(2010) using four planets (blue histogram) 
and six planets (red histogram). For com- 
parison we also show the Gaussian with \i = 
and a 2 = 1. As the number of data 
points is large (N = 241), there is no dif- 
ference between this Gaussian and Student's 
t-distribution. 
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4. Repeat steps 1 to 3 from n = 1 to n = N 
and compute the goodness of the predic- 
tion of the whole data set by multiplying 
the likelihoods obtained in step 3. 

Steps 3 and 4 require the data's error distribution to be known in order to evaluate the good- 
ness of the prediction for the left-out data point through its likelihood. For instance, if the 



data's errors are Gaussian, the goodness of the prediction is simply given by Eq. (13) as 
usual. Evidently, repeating steps 1 to 3 iV times is what makes leave-one-out cross-validation 
computationally expensive. It is also possible to leave out more than one data point in each 
stepj^] However, if the given data set is very small, cross-validation becomes unstable. A nice 



application of cross-validation can be found, e.g., in Hogg (2008) 



4.3 Bootstrapping 

Bootstrapping is somewhat more general than cross-validation, meaning it requires less knowl- 
edge about the origin of the data. Cross-validation requires the data's error distribution to be 
known in order to evaluate the likelihoods, whereas bootstrapping does not. Of course, this is 
an advantage if we do not have this knowledge. However, if we do know the data's errors, we 
should definitely exploit this knowledge by using cross-validation. Bootstrapping is discussed 
in Andrae (2010) in the context of error estimation. Therefore, we restrict its discussion here 
on the context of model comparison. 

Let us suppose we are given 4 measurements 2/1,2/2,2/3,2/4- We then draw subsamples of 
size 4 from this data set with replacement. These subsamples are called bootstrap samples. 
Examples are: 

• 2/1,2/2,2/3,2/4 itself, 

• 2/1,2/2,2/2,2/4, 

• 2/2,2/4,2/4,2/4, 

• 2/1,2/1,2/1,2/1, 

• etc. 

We draw a certain number of such bootstrap samples, and to every such sample we then fit all 
the models that are to be compared. 

In the context of model comparison, bootstrapping is typically used as "leave-one-out boot- 



strap" (e.g. Hastie et al. 2009). The algorithm is given by: 

1. Draw a certain number of bootstrap samples from a given data set. 

2. Fit all the models to every bootstrap sample. 

3. For the n-th data point y n in the given data set, consider only those bootstrap samples 
that do not contain y n . Predict y n from the models fitted to these bootstrap samples. 

4. Repeat step 3 from n = 1 to n = N and monitor the goodness of the predictions, e.g., by 
least squares. 

Like cross-validation, bootstrapping aims at the prediction error of the model. Therefore, it is 
sensitive to over- and underfittings. 



10 The reason why cross-validation is so reliable is that it draws on the predictive error of the model, rather 
than the fitting error. Therefore, cross-validation can detect underfitting (the model is not flexible enough to 
describe the data) and also overfitting (the model is too flexible compared to the data). The fitting error is only 
sensitive to underfitting, but not to overfitting (x 2 always decreases if the model becomes more complex). 
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5 Conclusions 

We have argued that there are two fundamental problems in using x 2 e d-> which are completely 
independent of each other: 

1. In Sect. |2j we have seen that estimating the number of degrees of freedom, which is 
necessary for evaluating xj? ed , is absolutely nontrivial in practice: 

• Concerning linear models, for N given data points and P fit parameters the number 
of degrees of freedom is somewhere between N — P and N — 1, where it is iV — P if 
and only if the basis functions of the linear model are linearly independent for the 
given data. Equation ^ provides a quantification for the effective number of fit 
parameters of a linear model. Priors can cause a linear model to become nonlinear. 

• Concerning nonlinear models, the number of degrees of freedom is somewhere be- 
tween zero and N — 1 and it may not even be constant during a fit, i.e., iV — P is a 
completely unjustified guess. The authors are not aware of any method that reliably 
estimates the number of degrees of freedom for nonlinear models. Consequently, it 
appears to be impossible to compute x 2 e d m this case - 



2. In Sect. [3j we have seen that the actual value of x 2 e d * s uncertain. If the number N of 
given data points is large, the uncertainty of x 2 e d * s approximately given by the Gaussian 
error a = ^/2/N. For N = 1, 000 data points, this means that within the 3cx-interval 
0.865 < Xred — 1-135 we cannot compare models or assess convergence. 

Given these considerations, it appears highly questionable whether the popularity of x 2 e d " 
which is certainly due to its apparent simplicity - is indeed justified. As a matter of fact, x 2 e d 
cannot be evaluated for a nonlinear model, because the number of degrees of freedom is unknown 
in this case. This is a severe restriction, because many relevant models are nonlinear. Moreover, 
even for linear models, Xred nas to be used with due caution, considering the uncertainty in its 
value. 

Concerning alternative methods for model comparison, we have explained cross-validation 
and bootstrapping in Sect. |4j We also explained how the normalised residuals of a model can be 
used to infer how close this model is to the true model underlying the given data. Concerning 



alternative methods for error estimation, we refer the interested reader to Andrae (2010). 

Finally, we want to emphasise that the above considerations concerning x 2 e d nave no impact 
on the correctness of minimising a x 2 m order to fit a model to data. Fitting models to data is 
a completely different task that should not be confused with model comparison or convergence 
testing. Minimising x 2 is the correct thing to do whenever the data's measurement errors are 
Gaussian and a maximum-likelihood estimate is desired. 
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